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ABSTRACT 



This paper tackles 

the problem of understanding whether Self-Regulated Learning (SRL) is taking place and whether students are developing SRL abilities in a Computer 
Supported Collaborative Learning (CSCL) environment. To this end, the authors propose a taxonomy of indicators based on Zimmerman’s learning cycle 
model, with several adaptations derived from subsequent research on SRL in Technology Enhanced Learning Environments. These indicators support the 
identification of SRL-related events in CSCL processes and have been tested in an experimental setting (the online component of a blended course for 
teachers) to analyse the messages exchanged by the learners. Content analysis was carried out manually by two coders on a sample of messages. The results 
of this descriptive, exploratory study are illustrated and compared with those previously obtained with different methods. 
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INTRODUCTION 

Research in the field of Self-Regulated Learning (SRL) covers the pedagogical, social, emotional, motivational, cognitive 
and meta-cognitive aspects involved when students leam to control their own learning processes. 

Computer supported collaborative learning (CSCL), on the other hand, is based on socio-constructivist theories of learning 
and concerns the implementation of collaborative learning in virtual communities. In this approach, computer mediated 
communication is used to allow group interactions at a distance, mostly based on textual, asynchronous communication 
among trainees, suitably scaffolded by tutors. 

In CSCL, SRL competence and, in particular, meta-cognitive skills are often among the explicit or implicit objectives of the 
learning process. This is due, first of all, to the fact that many learners are new to this approach and therefore lack some of 
the meta-cognitive skills that are needed to take full advantage of the training method. As a consequence, SRL is often 
required but also fostered by CSCL initiatives. Moreover, the very nature of CSCL learning processes is mostly based on 
textual interaction, and this makes it particularly well suited to reflection not only on content knowledge but also on the 
learning process itself. Since meta-cognition is an important component of SRL, CSCL is regarded as a promising field for 
its development. 

Research in this area has so far relied on interviews where learners are requested to describe, ex-post, the strategies and the 
methods they used during the learning process, or questionnaires aimed at eliciting the learners’ opinions about the whole 
process. However, according to many researchers, interaction analysis is another important source of information about the 
learning dynamics, and it can be quite rich, especially when meta-reflection activities receive explicit emphasis in the 
learning process. This paper proposes an exploratory study where a set of indicators of SRL in CSCL processes is used to 
guide quantitative content analysis of a sample of interactions between a group of trainee teachers during an online course in 
Educational Technology. The outcomes of the study are discussed and compared to those that had been previously obtained 
with classical methods. Aim of this preliminary, exploratory study is to evaluate the feasibility, reliability and cost- 
effectiveness of the approach proposed in view of a possible replicability on a larger scale. 

METHOD 



Theoretical Bases 

In order to define indicators of the practice, and possibly development, of SRL abilities, we started from the work carried out 
by Zimmermann (2000) on SRL and further elaborated by Carneiro et al. (2005), Willem et al (2006) and Banyard et al 
(2006) concerning the potential of Technology Enhanced Learning Environment for SRL development. 

As a matter of fact, the “process” and the “component” model of SRL, clearly suggest to take into consideration: 

• the learners’ abilities to plan, monitor and evaluate their own learning process; these can be investigated by looking for 
clues about the learners’ active contribution to choosing learning objectives and contents, decide or adapt learning 
strategies and suitably configure their learning environment, evaluate their learning results by comparing their outcomes 
with those of their peers and with models provided (the “process” model is seen as consisting of planning, monitoring 
and evaluation phases); 



• the learners’ abilities to cope with cognitive, meta-cognitive, emotional and motivational challenges imposed by the 
learning process; these can be captured by identifying cues that demonstrate awareness and application of strategies to 
solve complex problems, to cope with stress and anxiety, to keep up motivation, to relate with peers in a smooth and 
profitable way (the “component” model sees SRL as consisting of cognitive and meta-cognitive skills, as well as 
motivational and emotional aspects); 

• the learners’ abilities to practice the above both in individual study and in a collaborative learning context, be it face-to- 
face or at a distance. 

The indicators of SRL abilities proposed in this paper derive from the above theoretical framework and are shown in Table 1. 
This table specifies what should be observed into students messages in order to support the claim that some kind of self- 
regulation has been taking place. The underlying assumption of the study is that if a message contains reference to self- 
regulated actions then we can assume that the sender, having taken those actions, has practised self-regulation to some 
extent. The opposite, however, can not be claimed, in that if a student never expresses in his/her messages some self- 
regulation activity this doesn’t mean that self-regulation has not occurred. 



Procedure 

The complex nature of SRL skills suggests to adopt a mixed approach (qualitative and quantitative) to gather information 
about the application and development of SRL from the analysis of interactions between participants in an online course. 

Content analysis of interactions seems to have the required features, because it combines qualitative analysis of individual 
messages with quantitative elaboration of its results. This technique has been increasingly used by researchers to gain insight 
about CSCL dynamics, taking advantage of the permanent nature of computer conference transcripts. It can be used to study 
both the manifest and the latent content of transcripts. Of course, the manifest content can be investigated with more 
objectivity although, we agree with Rourke (2001), that often the most interesting research questions require the analysis of 
latent variables. The practice of SRL abilities and - let alone- their development, definitely belong to those “latent projective 
variables” whose “locus shifts to the coders interpretations of the meanings of the content” (Rourke, 2001). 

The identified indicators have been experimentally used to analyse the learning dynamics that took place in two activities of 
the online component of a blended teacher training course in educational technology. This course was run in 2005 by ITD- 
CNR for the Specialisation School for Secondary Education of the Liguria region (Delfino, Manca, Persico, in print). The 
course lasted 12 weeks and involved 95 students and 8 tutors who exchanged, in total, 7605 messages. Among these, the 
students messages were around 77% of the total. The two activities selected for this study were carried out by one sub-group 
of eight students and one tutor, they lasted about six weeks (three weeks each), for a total of 249 messages exchanged. The 
sample chosen has similar characteristics to the whole cohort of students: same ratio between males and females, same 
mixture of backgrounds, average grade in final assessment very similar to the average grade of all the students (see Table 2 
for data about the sample). Both activities were intended to be collaborative: the first was a role play, where students were 
required to find assets and weaknesses of a WebQuest (Dodge, 1997), the second was a case study on school based learning 
communities. Two coders were employed, who both examined all the students messages of the sample. One coder was one 
of the course tutors, while the other was an external rater. After coding, the inter-rater reliability was calculated, in terms of 
percent agreement. More accurate measures of the inter-rater reliability were not deemed to be worth using, given the 
exploratory nature of this study. However, should the study be extended to a bigger sample of messages, more adequate 
measures will be used to account for chance agreement (De Wever et al, 2005). Following the computation of the inter-rater 
reliability, the coders discussed the controversial cases until they reached 100% agreement. The reported data refer to these 
cases. 



Table 2 Features of the sample of messages analysed 





Stud, msgs 


Tutor msgs 


Total 

msgs 




mean 


SD 


N 


% 


N 


Activity 1 


11,3 


5,4 


14 


13,5 


104 


Activity2 


11,1 


17,1 


17 


11,7 


145 



The chosen unit of analysis was the message. This choice derived by the fact that messages are objectively identifiable, they 
are determined by the message author and produce a large, yet manageable set of cases. Furthermore, they turned out to 
exhaustively contain the indicators proposed in Table 1. However, they have a drawback: long messages sometimes contain 
more than one occurrence of our indicators. Although in most cases one message contains one or no occurrences, there are 
exceptions, and this makes some of the data more difficult to interpret. 



Some quantitative data about the two activities were also considered, such as the number of messages exchanged per day and 
the contribution of individual students to the discussion. These data helped us gain a global picture of learning dynamics in 





the considered activities, although they did not provide much information about the development of self-regulation, safe a 
generic indication of the phases where more support from the tutors was needed. 



Table 1 - A taxonomy of indicators of self-regulation 







Planning 


Execution and Monitoring 


Evaluation 


cognitive 
and meta- 
cognitive 


individual 


Code: PCI 

- Making plans on how to proceed 
in the learning process: breaking 
up tasks in sub-tasks, establishing 
deadlines, detecting priorities, etc. 

- Detecting plan changes necessary 
to overcome failures. 


Code: MCI 

- Enact plans. 

Work consistently on the 
assigned task. 

- Monitoring plan fulfilment. 

- Making syntheses of the work 
done and objectives reached. 


Code: MCI 

- Assessing own learning. 

Analysing results, spotting 
difficulties and causes of failures. 

- Reflecting on individual learning 
achieved. 

- Comparing one’s work with that 
of peers 


social 


Code: PCS 

- Making proposals on how to 
proceed in the learning process. 

- Discussing and negotiating on 
planning aspects, 

- Detecting together plan changes 
necessary to overcome failures. 

- Encouraging/requesting others to 

express their opinions 


Code: MCS 

- Quoting peers contributions, 
asking questions, reacting to and 
mediating among peers. 

- Checking understanding 

- Summarising the ideas suggested 
by all group members. 

- Encouraging peers to act. 


Code: ECS 

- Assessing group learning. 

- Commenting group 
achievements. 

- Reflecting on group learning. 


motivational 

and 

emotional 


individual 


Code: PMI 

- Exploring one’s expectations 
about the current learning activity. 
- Anticipating possible emotional 
aspects. 


Code: MMI 

- Expressing one’s emotions and 

motivations 

- Looking for appropriate support 

when needed 


Code: EMI 

- Comparing one’s current 
motivation and emotions with the 
original ones. 

- Understanding the reasons of 
possible changes to plans. 

- Commenting on emotional 
aspects developed during the 
learning process 


social 


Code: PMS 

- Discussing expectations and 
motivations about the current 
learning activity and learning in 
general. - Sharing motivations for 
own commitment. 


Code: MMS 

- Encouraging peers to express 
their emotions and motivations. 
- Disclosing oneself to peers, 
encouraging and providing 
emotional support to peers. 


Code: EMS 

- Expressing appreciation for 
peers’ efforts, contributions and 
results. 

- Spotting group’s malfunctioning. 



FINDINGS 

Table 3 gives an idea of the replicability of the approach: inter-rater reliability, in terms of percent agreement, was quite 
acceptable, while the percentage of significant messages was not very high, which is coherent with the fact that SRL can be 
investigated by using a set of latent projective variables, that do not reveal themselves systematically. 

Table 3 Sample features in term s of coding results 





Meaningful messages 


Inter-rater 

reliability 




N 


% 




Activity 1 


32 


35 


88,6 


Activity2 


49 


38 


80,0 























Number of messages per category 
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□ Activity 1 
■ Activity 2 



Fig. 1 Summary of coding results 

The main results of the content analysis are reported in Fig.l to 4. 

These data show that trainees participated more to Activity 2 (the case study) than to Activity 1 (the role play). This is true, 
not only in terms of mere number of messages, but also in terms of “SRL density”. In fact, the two activities had the same 
duration, but the percentage of messages that were found significant through content analysis is higher in the second. This 
can partly be explained by the nature of the two activities, but it can also support the hypothesis that the use of SRL abilities 
increased. Most likely, both explanations are true. Activity 1, in fact, being a role play, proposed an inherent plan (who does 
what and with what role) that relieved students from some planning tasks (this is confirmed by the data in Fig. 2, where 
indicators of planning events in Activity 1 are significantly less than those of Activity 2). However, Activity 2 also shows a 
generally higher concentration of SRL-related events, therefore supporting the idea that students self-regulated their learning 
better in this phase of work, as concerns not only planning, but also monitoring and evaluation tasks. 





Message categorisation according to "process" 
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Fig. 2 Coding results along the categories of the process model 



Message categorisation "individual" vs "social" 



60 - 
50 - 




□ Activity 1 
■ Activity 2 I 



Individual 



Social 



Fig. 3 Coding results along the individual vs social categories 



Message categorisation: cognitive and 
metacognitive vs emotional and motivational 
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Fig.4 Coding results along the categories cognitive and meta-cognitive vs. emotional and motivational 

Fig. 2 shows that indicators related to SRL at a social level were definitely more frequent than indicators showing SRL of the 
individual. Again, there are two possible reasons behind this and it is likely that both have concurred to determine our data. 
One reason is that CSCL strategies tend to favour the social aspects of SRL more than the individual aspects (for example, 
students feel encouraged to plan, monitor and evaluate the group work, more than they do with their own individual work). 
The second explanation is that in a CSCL environment students feel the need to express the social aspects of SRL in their 
messages more than they do with the individual aspects. In other words, they might be planning, monitoring and evaluating 
their own individual work as well, but they do not feel so much the need to write it in their messages. In any case, this result 
apparently confirms the outcomes of a previous study where a different method was used to investigate SRL development in 
the same course (Dettori, Giannetti, & Persico, 2006). The above paper presents the results of a survey carried out with two 
questionnaires: one was filled in by SRL experts and another by 72 of the 95 trainees taking part in this course. Both 
concerned the interviewees’ opinions about the support received in practicing SRL during the course. The survey showed 
that the potential of the environment used was deemed valuable especially as concerns the social aspects of SRL: students, as 
a matter of fact, claimed they felt a strong social support to their own SRL development from tutors and, even more, from 
peers. 

The same study reported that, according to SRL experts, the emotional and motivational components of such support were 
stronger than the cognitive/meta-cognitive ones. On the other hand, according to the trainees, the former was weaker than the 
latter. The present study, and in particularly the data shown in Fig.4, seem to confirm the students opinion. As a matter of 
fact, the indicators of cognitive and meta-cognitive aspects were more frequent than the emotional and motivational ones in 
both activities. 



DISCUSSION 

Evidence of the presence of one indicator will obviously not - per se - prove the development of SRL. It will only support 



the claim that that particular aspect of SRL was practiced. However, Zimmerman’s (1998) studies on SRL suggest that these 
abilities develop through social support and practice. In addition, increased frequency of the indicators during the learning 
process can be regarded as a significant clue of SRL development. The opposite, however, is not necessarily true. The fact 
that SRL indicators are not to be found in students messages doesn’t necessarily mean that the students did not control their 
learning: they might simply have not made the process explicit in their messages. So, why should one bother with content 
analysis to find out? This study is mostly aimed at understanding whether this method can provide significant information 
that should be regarded as complementary to what can be obtained with other methods. In general, information about SRL 
abilities are sought after through interviews with the subjects, questionnaires or observation. Questionnaires and interviews 
give us the opinions of the learners, or other information that are reported by them. On the other hand, observation and 
content analysis of messages exchanged allow us to analyse directly what students actually did. They do not give us access to 
all that has taken place in their heads, but they allow us to work on data that have not been filtered by the learners and their 
opinions. It is on this bases that we believe it is useful to take both sources of data into consideration. 

This is only an exploratory study. It was carried out on a small sample, with a manual method, little statistics. The aims were: 

to find out whether content analysis with these indicators provides data consistent with previous research, to understand 
whether it is cost effective and if the indicators are sufficiently well-defined to assure an acceptable reliability, to refine the 
indicators and verify whether there are ways to partially automate the textual analysis process. 

As for the first point, according to the collected data the cost-effectiveness of the approach is encouraging enough to plan an 
extension of the study to a wider sample and to carry out replication studies in different contexts. 

While the answers to the first point are quite satisfactory, the second point is a bit controversial. The inter-rater reliability, in 
fact, turned out to be pretty good (at least, percent agreement is acceptable, but for bigger samples it would be worthwhile to 
use more sophisticated measures of reliability such as Kohen K) (Capozzoli et al, 1999). However, the percentage of 
meaningful messages is not very high, and this makes the raters’ work not so cost effective. As for the third point, again, 
there are good news and bad news. The good news is that the indicators table turned out to be quite complete: some 
indicators were added thanks to the suggestions of the raters, but the overall structure and most of the original indicators 
were fit to the purpose. The bad news is that there doesn’t seem to be an easy way to automate the analysis process. In fact, 
while in some cases content analysis can be made easier by software tools that look for typical expressions that consistently 
introduce the clues that are searched for, in the case of SRL there doesn’t seem to be any typical expression that introduce the 
sentences we are looking for. Planning actions can be introduced by “I propose...”, “Why don’t we...”, “Couldn’t we...” and 
too many more expressions (or their Italian equivalent). The same holds for monitoring and evaluation sentence patterns: 
there are simply too many ways to introduce a sentence where monitoring or evaluation considerations are brought forward. 

To conclude, SRL development can be revealed by a set of “latent projective variables”, and the proposed set of indicators, 
derived from widely accepted models from the SRL literature, seems to work properly to this purpose. However, there are 
some important caveat. Firstly, we acknowledge that the use of this kind of variables makes content analysis an inherently 
subjective and interpretative process. Secondly, researchers who intend to use this method should be aware that what can be 
found in messages is likely to be true, but not the other way around. 
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